Data Processing System for Guidance, Control, and Testing Autonomous Vehicle Features and Driver Response

ABSTRACT

Systems and methods are disclosed for guidance, control, and testing of autonomous vehicle features and a driver&#39;s response thereto. The system may activate a plurality of autonomous driving features of an autonomous vehicle. In response to a determination to initiate a driving test, the system may generate an indication to a driver of the autonomous vehicle of the initiation of the driving test and may deactivate or adjust parameters of one or more of the plurality of autonomous driving features. The system may receive, from one or more sensors of the autonomous vehicle or one or more sensors of a mobile computing device within the autonomous vehicle, driving data associated with the autonomous vehicle. Based on the driving data associated with the autonomous vehicle, the system may determine the driver&#39;s response time and actions taken by the driver during the driving test. Moreover, in response to a determination to end the driving test, the system may reactivate the one or more of the plurality of autonomous driving features previously deactivated or may readjust previously adjusted parameters. In some aspects, based on the driver&#39;s response time and actions taken by the driver during the driving test, a drive score may be generated for the driver.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 15/437,988, filed Feb. 21, 2017 and entitled “Data ProcessingSystem for Guidance, Control, and Testing Autonomous Vehicle Featuresand Driver Response.” The prior application is incorporated herein byreference in its entirety.

TECHNICAL FIELD

Aspects of the disclosure generally relate to a system and method forguidance, control, and testing of autonomous vehicle features and adriver's response thereto.

BACKGROUND

For some autonomous driving levels (e.g., semi-autonomous driving), thedriver may be expected to remain attentive and alert and be able to takeover driving at a moment's notice. There may be an increased risk of avehicle accident if the driver is not able to take over control of thevehicle, assess the environment, and react appropriately within theparameters identified by a manufacturer of an autonomous driving featureor autonomous vehicle.

SUMMARY

The following presents a simplified summary in order to provide a basicunderstanding of some aspects of the disclosure. The summary is not anextensive overview of the disclosure. It is neither intended to identifykey or critical elements of the disclosure nor to delineate the scope ofthe disclosure. The following summary merely presents some concepts ofthe disclosure in a simplified form as a prelude to the descriptionbelow.

Aspects of the disclosure relate to a system comprising an autonomousvehicle having one or more sensors configured to measure driving dataassociated with the autonomous vehicle. The autonomous vehicle maycomprise a transmitter configured to transmit driving data measured bythe one or more sensors, a first processor, and first memory. The firstmemory of the autonomous vehicle may store computer-executableinstructions that, when executed by the first processor, cause the firstprocessor of the autonomous vehicle to activate a plurality ofautonomous driving features of the autonomous vehicle. In response to adetermination to initiate a driving test, the autonomous vehicle maygenerate an indication to a driver of the autonomous vehicle of theinitiation of the driving test and deactivate or adjust parameters ofone or more of the plurality of autonomous driving features. Drivingdata associated with the autonomous vehicle may be received from the oneor more sensors of the autonomous vehicle. The autonomous vehicle maysend, via the transmitter, the driving data associated with theautonomous vehicle to an autonomous vehicle drive test computing device.

In some aspects, the autonomous vehicle drive test computing device maycomprise a receiver, a second processor, and second memory. The secondmemory may store computer-executable instructions that, when executed bythe second processor, cause the second processor of the autonomousvehicle drive test computing device to receive, using the receiver ofthe autonomous vehicle drive test computing device, the driving dataassociated with the autonomous vehicle. Based on the driving dataassociated with the autonomous vehicle, the autonomous vehicle drivetest computing device may determine the driver's response time andactions taken by the driver during the driving test. In response to adetermination to end the driving test, the autonomous vehicle mayreactivate the deactivated or readjust the adjusted parameters of one ormore of the plurality of autonomous driving features. The determinationto end the driving test may comprise a determination that the drivercompleted the driving test or a determination that the driver's responsetime exceeds a threshold time

In some aspects, the determination to initiate the driving test may bebased on one or more of a comparison of a current time and a timeassociated with the driving test or a location of the autonomousvehicle. The plurality of autonomous driving features may comprise atleast two of autonomous speed control, autonomous steering, orautonomous braking. The driving test may comprise a test to merge on toa road, and the deactivated or adjusted parameters of one or more of theplurality of autonomous driving features may comprise at leastautonomous lane changing and autonomous speed control. Alternatively,the driving test may comprise a test to maintain a safe distance fromanother vehicle in front of the autonomous vehicle, and the deactivatedor adjusted parameters of one or more of the plurality of autonomousdriving features may comprise at least autonomous speed control.

In some aspects, the second memory may store additionalcomputer-executable instructions that, when executed by the secondprocessor, causes the second processor of the autonomous vehicle drivetest computing device to generate a data log comprising an indication ofthe driving test, the driver's response time, and the actions taken bythe driver during the driving test. In some aspects, the second memorymay store additional computer-executable instructions that, whenexecuted by the second processor, causes the second processor of theautonomous vehicle drive test computing device to, based on the driver'sresponse time and actions taken by the driver during the driving test,generate a drive score for the driver and/or adjust a property ofinsurance for the driver or the autonomous vehicle.

Other features and advantages of the disclosure will be apparent fromthe additional description provided herein.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present invention and theadvantages thereof may be acquired by referring to the followingdescription in consideration of the accompanying drawings, in which likereference numbers indicate like features, and wherein:

FIG. 1 is a diagram illustrating various example components of a systemfor guidance, control, and testing of autonomous vehicle features and adriver's response thereto according to one or more aspects of thedisclosure.

FIG. 2 is a flow diagram illustrating an example method of guidance,control, and testing of autonomous vehicle features and a driver'sresponse thereto according to one or more aspects of the disclosure.

FIG. 3 is a flow diagram illustrating an example method of guidance,control, and testing of autonomous vehicle features and a driver'sresponse thereto according to one or more aspects of the disclosure.

FIG. 4 illustrates a network environment and computing systems that maybe used to implement aspects of the disclosure.

DETAILED DESCRIPTION

In the following description of the various embodiments, reference ismade to the accompanying drawings, which form a part hereof, and inwhich is shown by way of illustration, various embodiments of thedisclosure that may be practiced. It is to be understood that otherembodiments may be utilized.

As will be appreciated by one of skill in the art upon reading thefollowing disclosure, various aspects described herein may be embodiedas a method, a computer system, or a computer program product. Aspectsmay take the form of a computing device configured to perform specifiedactions. Furthermore, such aspects may take the form of a computerprogram product stored by one or more computer-readable storage mediahaving computer-readable program code, or instructions, embodied in oron the storage media. Any suitable computer readable storage media maybe utilized, including hard disks, CD-ROMs, optical storage devices,magnetic storage devices, and/or any combination thereof. In addition,various signals representing data or events as described herein may betransferred between a source and a destination in the form ofelectromagnetic waves traveling through signal-conducting media such asmetal wires, optical fibers, and/or wireless transmission media (e.g.,air and/or space).

FIG. 1 is a diagram illustrating various example components of a system100 for guidance, control, and testing of autonomous vehicle featuresand a driver's response thereto according to one or more aspects of thedisclosure. The system 100 may include a plurality of vehicles, such asa vehicle 110 and other vehicles (not illustrated), one or more mobiledevices 130 (e.g., a mobile device of a driver outside of the vehicle),one or more driving data sources 140, an autonomous vehicle drive testserver 150, another computing device 160, and additional relatedcomponents. Each component of the system 100 may include a computingdevice (or system) having some or all of the following structuralcomponents.

For example, the drive test computer or computing device 151 may have aprocessor for controlling overall operation of the computing device 151and its associated components, including RAM, ROM, input/output module,and memory. The computing device 151, along with one or more additionaldevices (e.g., vehicle 110, mobile device 130, driving data source 140,computing device 160), may correspond to any of multiple systems ordevices, such as autonomous vehicle drive test computing devices orsystems, configured as described herein for transmitting and receivingsensor data, initiating a drive test (e.g., by deactivating or adjustingparameters of one or more autonomous features), and measuring thedriver's response to the drive test. Sensor data can include datacollected from mobile devices (e.g., a driver's mobile phone, apassenger's mobile phone, a pedestrian's mobile phone), vehicle sensors,on-board diagnostic (OBD) systems, and/or off the shelf devices. Sensordata may refer to information pertaining to one or more actions orevents performed or observed by a vehicle, a pedestrian, or trafficdevices and can include aspects of information identified or determinedfrom data collected from a vehicle, a stationary device, or a mobiledevice. Sensor data can include, for example, location data, speed orvelocity data, acceleration data, presence data, time data, directiondata, mobile device orientation data, rotation/gyroscopic data, and thelike.

The drive test computer 151 may include an Input/Output (I/O) modulehaving a microphone, keypad, touch screen, and/or stylus through which auser of the computing device 151 may provide input, and may also includeone or more of a speaker for providing audio input/output and a videodisplay device for providing textual, audiovisual and/or graphicaloutput. Software may be stored within the memory of the drive testcomputer and/or other storage to provide instructions to its processorfor enabling device 151 to perform various functions. For example, thecomputing device's memory may store software used by the device 151,such as an operating system, application programs, and an associatedinternal or external database 152. The memory unit may include one ormore of volatile and/or non-volatile computer memory to storecomputer-executable instructions, data, and/or other information. Theprocessor of the computer 151 and its associated components may allowthe drive test computer 151 to execute a series of computer-readableinstructions to transmit or receive sensor data, process sensor data,initiate one or more drive tests, and determine the driver's responsethereto. One or more application programs used by the drive testcomputing device 151 may include computer executable instructions (e.g.,autonomous vehicle drive test algorithms, and the like) for providingautonomous vehicle drive tests and performing other related functions asdescribed herein.

The drive test computing device 151 may operate in a networkedenvironment supporting connections to one or more remote computers, suchas various other terminals/devices (e.g., terminal 160, which mayinclude a display). The drive test computing device 151, and the relatedterminals/devices, may communicate with devices installed in vehicles,mobile devices that may travel within vehicles, or devices outside ofvehicles. Thus, the drive test computing device 151 and its associatedterminals/devices may each include personal computers (e.g., laptop,desktop, or tablet computers) and/or servers (e.g., web servers,database servers) and may communicate with vehicle-based devices (e.g.,on-board vehicle computers, short-range vehicle communication systems,sensor and telematics devices), or mobile communication devices (e.g.,mobile phones, portable computing devices, and the like).

The devices illustrated in system 100 may communicate via networkconnections depicted such as a local area network (LAN) and a wide areanetwork (WAN), and a wireless telecommunications network, but may alsoinclude other networks. When used in a LAN networking environment, thedrive test computing device 151 may be connected to the LAN through anetwork interface or adapter. When used in a WAN networking environment,the device 151 may include a modem or other means for establishingcommunications over the WAN, such as a network (e.g., the Internet).When used in a wireless telecommunications network, the device 151 mayinclude one or more transceivers, digital signal processors, andadditional circuitry and software for communicating with wirelesscomputing devices (e.g., mobile phones, short-range vehiclecommunication systems, vehicle sensing and telematics devices) via oneor more network devices (e.g., base transceiver stations) in thewireless network. It will be appreciated that the network connectionsshown and described above are illustrative and other means ofestablishing a communications link between the computers may be used.The existence of any of various network protocols such as TCP/IP,Ethernet, FTP, HTTP and the like, and of various wireless communicationtechnologies such as GSM, CDMA, Wi-Fi, and WiMAX, is presumed, and thevarious computing devices and drive test system components describedherein may be configured to communicate using any of these networkprotocols or technologies.

The vehicle 110 may be, for example, an automobile, motorcycle, scooter,bus, recreational vehicle, boat, or other vehicle for having autonomousfunctionality (e.g., semi-autonomous levels 1, 2, or 3 or fullyautonomous). A mobile computing device 116 within the vehicle 110 may beused to collect sensor data (e.g., via sensors 118), generate prompts(e.g., audio or visual) to the driver, receive input from the driver(e.g., spoken commands or touch inputs), and/or to receive sensor orother driving data from the vehicle 110 (e.g., via vehicle sensors 119).The mobile device 116 may process the driving data to measure thedriver's response to an autonomous vehicle test in the vehicle 110and/or transmit the data to the autonomous vehicle drive test server 150or other external computing devices. Mobile computing device 116 may be,for example, mobile phones, personal digital assistants (PDAs), tabletcomputers, laptop computers, smartwatches, and other devices that may becarried by drivers or passengers inside or outside of the vehicle 110.

The mobile computing device 116 may contain some or all of thehardware/software components of the computing device 151 describedabove. Software applications may be installed on and execute on themobile device 116. The software applications may be configured toreceive sensor data from internal sensors 118, such as acceleration,velocity, location, and the like and/or communicate with vehicle sensors119 or other vehicle communication systems to sense or receive sensordata. For example, mobile device 116 equipped with Global PositioningSystem (GPS) functionality may determine vehicle location, speed,direction and other basic driving data without needing to communicatewith vehicle sensors or external vehicle systems. The mobile softwareapplication may also receive sensor data from a wearable device, such asa smartwatch or a fitness band. In other examples, the softwareapplication on the mobile device 116 may be configured to receive someor all of the sensed data collected by sensors 119 of the vehicle 110.

When mobile computing device 116 within the vehicle 110 is used to sensevehicle data, the mobile computing device 116 may store, analyze, and/ortransmit the vehicle data to one or more other computing devices. Forexample, mobile device 116 may transmit vehicle data directly to vehicledrive test server 150, and thus may be used instead of sensors orcommunication systems of the vehicle 110.

The mobile device 116 may include various sensors 118 capable ofdetecting and recording conditions at and operational parameters of thevehicle 110 if the mobile device 116 is inside the vehicle. The sensors118 may be used to sense, for example, the location of the mobile device116, such as the GPS coordinates (e.g., latitude and longitude). Thelocation of the mobile device 116 may also be determined based onwireless networks the mobile device has connected to, such as Wi-Finetworks, cellular networks, and the like.

The sensors 118 of the mobile device 116, such as a GPS and/or acompass, may sense the speed and/or direction at which the mobile device116 and accordingly vehicle 110 is traveling. An accelerometer of themobile device 116 may sense the acceleration of the mobile device. Agyroscope may be used to determine the orientation of the mobile device.The gyroscope may also be used to measure the speed of rotation of themobile device 116. A magnetometer may be used to measure the strengthand direction of the magnetic field relative to the mobile device. Thesensors 118 previously described are exemplary, and the mobile device116 may include any other sensors used for autonomous vehicle drivetesting.

The data collected by the mobile device 116 may be stored and/oranalyzed within the mobile device 116. The processing components of themobile computing device 116 may be used to analyze sensor data and/ordetermine the driver's response to one or more driving tests.Additionally or alternatively, the mobile device 116 may transmit, via awired or wireless transmission network, the data to one or more externaldevices for storage or analysis, such as vehicle computer 114 or vehicledrive test server 150. In other words, mobile computing device 116 maybe used in conjunction with, or in place of, the vehicle computer 114 orvehicle drive test server 150 to measure the driver's response to adrive test.

The vehicle computer 114 of the vehicle 110 may contain some or all ofthe hardware/software components of the computing device 151 describedabove. The vehicle computer 114 may receive sensor data from the mobiledevice 116 and/or from sensors 119 built into the vehicle 110. Forexample, vehicle computer 114 may receive accelerometer data from themobile device 116 or an accelerometer in the vehicle 110. Sensors 119may include, for example, telematics devices integrated with the vehicleand/or aftermarket telematics devices. The telematics devices may beused to track location, vehicle diagnostics, speed, acceleration, andthe like. For example, the sensors 119 may be used to sense the locationof the vehicle 110, such as the GPS coordinates (e.g., latitude andlongitude). The location of the vehicle 110 may also be determined basedon wireless networks the vehicle 110 has connected to, such as Wi-Finetworks, cellular networks, and the like. In some aspects, theaftermarket telematics devices may be connected to the vehicle by wire,such as via the vehicle's OBD port.

The vehicle computer 114 may act as a gateway device between the mobiledevice 116 and the vehicle drive test server 150. For example, thevehicle computer 114 may receive sensor data (or data indicating thedriver's response to a test) from the mobile device 116 and forward thereceived data to the drive test server 150. The vehicle 110 may includea short-range communication system 112, which will be described infurther detail below.

The transceiver 112 may comprise or be part of a short-rangecommunication system, such as a vehicle-based data transmission systemconfigured to transmit vehicle data to other nearby vehicles, and toreceive vehicle data from other nearby vehicles. In some examples, thetransceiver 112 may use the dedicated short-range communications (DSRC)protocols and standards to perform wireless communications betweenvehicles. In the United States, 75 MHz in the 5.850-5.925 GHz band havebeen allocated for DSRC systems and applications, and various other DSRCallocations have been defined in other countries and jurisdictions.However, the communication system 112 need not use DSRC, and may beimplemented using other short-range wireless protocols in otherexamples, such as WLAN communication protocols (e.g., IEEE 802.11),Bluetooth (e.g., IEEE 802.15.1), or one or more of the CommunicationAccess for Land Mobiles (CALM) wireless communication protocols and airinterfaces.

The vehicle-to-vehicle (V2V) transmissions between the communicationsystem 112 and another vehicle's communication system may be sent viaDSRC, Bluetooth, satellite, GSM, LTE, infrared, IEEE 802.11, WiMAX,RFID, and/or any suitable wireless communication media, standards, andprotocols. In certain systems, the communication system 112 may includespecialized hardware installed in vehicle 110 (e.g., transceivers,antennas, etc.), while in other examples the communication system 112may be implemented using existing vehicle hardware components (e.g.,radio and satellite equipment, navigation computers) or may beimplemented by software running on the mobile device 116 of drivers andpassengers within the vehicle 110.

The range of V2V communications between vehicle communication systemsmay depend on the wireless communication standards and protocols used,the transmission/reception hardware (e.g., transceivers, power sources,antennas), and other factors. Short-range V2V communications may rangefrom just a few feet to many miles. V2V communications also may includevehicle-to-infrastructure (V2I) communications, such as transmissionsfrom vehicles to non-vehicle receiving devices, for example, tollbooths, rail road crossings, and road-side traffic monitoring devices.Certain V2V communication systems may periodically broadcast data from avehicle 110 to any other vehicle, or other infrastructure device capableof receiving the communication, within the range of the vehicle'stransmission capabilities. For example, a vehicle 110 may periodicallybroadcast (e.g., every 0.1 second, every 0.5 seconds, every second,every 5 seconds, etc.) certain vehicle data via its short-rangecommunication system 112, regardless of whether or not any othervehicles or reception devices are in range. In other examples, a vehiclecommunication system 112 may first detect nearby vehicles and receivingdevices, and may initialize communication with each by performing ahandshaking transaction before beginning to transmit its vehicle data tothe other vehicles and/or devices.

The types of vehicle data transmitted by the vehicle 110 may depend onthe protocols and standards used for the V2V communication, the range ofcommunications, and other factors. In certain examples, the vehicle 110may periodically broadcast corresponding sets of similar vehicle drivingdata, such as the location (which may include an absolute location inGPS coordinates or other coordinate systems, and/or a relative locationwith respect to another vehicle or a fixed point), speed, and directionof travel. In certain examples, the nodes in a V2V communication system(e.g., vehicles and other reception devices) may use internal clockswith synchronized time signals, and may send transmission times withinV2V communications, so that the receiver may calculate its distance fromthe transmitting node based on the difference between the transmissiontime and the reception time. The state or usage of the vehicle's 110controls and instruments may also be transmitted, for example, whetherthe vehicle is accelerating, braking, turning, and by how much, and/orwhich of the vehicle's instruments are currently activated by the driver(e.g., head lights, turn signals, hazard lights, cruise control, 4-wheeldrive, traction control, windshield wipers, etc.). Vehicle warnings suchas detection by the vehicle's 110 internal systems that the vehicle isskidding, that an impact has occurred, or that the vehicle's airbagshave been deployed, also may be transmitted in V2V communications.

The mobile computing device 116 may be used instead of, or inconjunction with, the communication system 112. For example, the mobiledevice 116 may communicate directly with the other vehicle or directlywith another mobile device, which may be inside or outside of the othervehicle. Additionally or alternatively, the other vehicle maycommunicate location information to vehicle 110, and vehicle 110 may inturn communicate this location information to the mobile device 116. Anydata collected by any vehicle sensor 119 or mobile device sensor 118 maybe transmitted via V2V or other communication to other nearby vehicles,mobile devices, or infrastructure devices receiving V2V communicationsfrom communication system 112 or communications directly from mobiledevice 116. Further, additional vehicle driving data not from thevehicle's sensors (e.g., vehicle make/model/year information, driverinformation, etc.) may be collected from other data sources, such as adriver's or passenger's mobile device 116, vehicle drive test server150, and/or another external computer system, and transmitted using V2Vcommunications to nearby vehicles and other transmitting and receivingdevices using communication system 112.

The system 100 may include one or more mobile computing device 130,which may be similar to mobile computing device 116, except that themobile computing device 130 may be outside of a vehicle (e.g., a mobiledevice of the driver while the driver is outside of the vehicle). Themobile computing device 130 may contain some or all of thehardware/software components of the computing device 151 describedabove. Software applications executing on the mobile device 130 may beconfigured to receive sensor data from sensors 132, such asacceleration, velocity, location, and the like. The location of themobile device 130 may be determined based on wireless networks themobile device has connected to, such as Wi-Fi networks, cellularnetworks, GPS, and the like. As described above, the sensors 132 mayalso comprise a compass, an accelerometer, a gyroscope, a magnetometer,or any other type of sensor. The mobile computing device 130 may alsocomprise other mobile devices or accessories, such as a wearable (e.g.,a smartwatch).

The system 100 may comprise one or more driving data sources 140. Eachdriving data source 140 may comprise one or more database storingdriving data, such as a particular driver's past driving telematicsdata, data indicative of driving situations likely to lead to an event,etc. Other types of driving data will be described in further detailbelow. Data from the driving data sources 140 may be accessed via, forexample, open application program interfaces (APIs), databases, softwaredevelopment kits (SDKs), V2V communication, and/or mobile device tomobile device communication.

In certain autonomous driving levels (e.g., U.S. Department ofTransportation's National Highway Traffic Safety Administration (NHTSA)autonomous levels 1, 2, or 3), the vehicle may take over theresponsibility of the driver in certain situations (which may beextended situations). For example, NHTSA level 1 may comprisefunction-specific automation, such as automation involving one or morecontrol functions. NHTSA level 2 may comprise combined functionautomation, such as automation of two or more primary control functionsworking together. NHTSA level 3 may comprise enabling the driver to cedecontrol of safety-critical functions (e.g., all safety-criticalfunctions) under specific driving conditions. The vehicle may rely onits sensors and internal systems to make driving decisions, includingcompliance with speed limits and other traffic rules, driver preferencesfor speed, staying in a lane, and managing safe distance from othervehicles in front of it. In instances at levels 3 and below, the drivermay be capable of taking over driving from the autonomous system at amoment's notice (and in some instances may be required by law to takeover driving). The driver may be expected to quickly gain awareness ofthe vehicle and the driving environment. In some situations, the vehiclemay announce to the driver for the driver to take over. If the driverdoes not take over driving when requested, the vehicle may enter afail-safe mode that stops the vehicle.

Vehicles described herein (and when driving in a particular autonomousmode, such as semi-autonomous mode) may alert the driver (e.g.,periodically) to assume control of the vehicle, such as full control ora lower level of control. In some aspects, the alert may be similar oridentical to a notification provided to the driver during an actualevent requiring or requesting the driver to take control. The driver mayindicate control by a sequence of events, such as grasping the steeringwheel and tapping on the brake pedal, or a similar sequence of inputsfrom the driver when the driver is properly positioned in the vehicle.

Systems and methods described herein may test and measure driverattentiveness while in an autonomous driving mode. In some aspects, thesystem may determine, based on the driver's response to the test, thatone or more autonomous features be disabled, the vehicle be locked outof one or more autonomous driving modes, and/or the vehicle be lockedout of operating altogether. Also described herein are systems andmethods for reporting attentiveness to an insurance company that isinsuring the vehicle and/or the driver and adjusting rates for thedriver and/or vehicle as a result of the measurements. Insurancecompanies may charge variable premiums based on measurable andquantifiable capabilities of a driver as it relates to skills foroperating an autonomous vehicle (e.g., level 1, 2, or 3) safely.

An advantageous benefit of the system and method described herein isthat drivers' skills may be kept sharp, even as autonomous features areused more and more. The ability of a person using his or her senses todetermine anomalies in a situation may decrease as occurrences of theanomalous situation decrease. This may be similar to the athlete wholoses a skill when the athlete has not practiced the skill for a longtime. When relying on autonomous driving, drivers responsible for beingable to retain control of a vehicle at a moment's notice may gainpractice in one or more aspects of this skill so that when the skill isneeded, the driver may have the necessary muscle memory ability.

Aspects related to the assessment, training, practice, and/or feedbackfor drivers driving autonomous vehicles (e.g., with certain autonomousfeatures activated and/or at a particular autonomous driving level, suchas level 1, 2, or 3) and quantitatively scoring driver performance ofthe driver's skills in order to adjust a property of insurance for thedevice (e.g., insurance premiums) based on the driver score aredescribed herein. These (and other aspects) will be described in furtherdetail below.

FIG. 2 is a flow diagram illustrating an example method of guidance,control, and testing of autonomous vehicle features and a driver'sresponse thereto according to one or more aspects of the disclosure. Thesteps illustrated in FIG. 2 may be performed by one or more of thedevices illustrated in FIG. 1. In some aspects, autonomous driving testsmay be customized for each user based on user input, driving history,and other received data.

In step 205, a computing device may generate a prompt requesting a userselection of one or more autonomous driving features to test. The promptmay present a plurality of test options to the user so that the user mayprovide his or her preference and/or select the types of tests that thedriver would like to engage in. In some aspects, the prompt may presentoptions for autonomous driving features of the vehicle to test (e.g.,autonomous features to deactivate or adjust the parameters of whiledriving in an autonomous driving mode, such as semi-autonomous driving,completely autonomous, or a different level of autonomous driving).Non-limiting examples of autonomous driving include autonomous speedcontrol, autonomous steering, autonomous braking, etc. For example, theoptions for autonomous features to deactivate or adjust the parametersof may include cruise control, brake support, lane keeping, lanechanging, obstacle detection and reaction, driving condition detection,automatic merging (e.g., while entering or exiting a freeway),self-parking, automatic lighting, and/or any other autonomous vehiclefeature.

In some aspects, the prompt may present options for types of tests thatthe driver would like to perform, and each type of test may comprise acombination of one or more autonomous driving features. For example, onetype of test may comprise a test to merge on to a freeway. In this test,the vehicle may deactivate (or adjust parameters of), for example,autonomous lane changing and/or autonomous speed control for theduration of the test so that the driver can practice merging on to thefreeway. Another type of test may comprise a test to maintain a safedistance from another vehicle. In this test, the vehicle may deactivate(or adjust parameters of), for example, autonomous speed control so thatthe driver can practice maintaining a certain distance from the vehiclein front. Other types of tests may comprise testing the ability to steeraround a bend, the ability to react to road announcements (e.g., laneclosed ahead), simulated vehicle faults, and the like.

In some aspects, selection of one or more autonomous driving features totest may be automatically performed by the computing device and/orsystem rather than a user selection. For example, the computing devicemay analyze driving data and may select the tests that will improve thedriver's abilities. The driver may be scored on the driver's ability, aswill be described in further detail below. The driver may review thetest results (e.g., offline) and may learn how the driver compares inthe test to a larger population of drivers and how the score may affectthe driver's insurance.

The prompt may comprise a visual prompt, such as a graphical userinterface (GUI) display on a touchscreen display, a non-touchscreendisplay, a head-up display in a vehicle 110, a virtual or augmentedreality display, or any other type of display device. The prompt mayadditionally or alternatively comprise an audio prompt, such as spokenquestions or instructions presented via a speaker of the vehicle or amobile device 116 or 130 (e.g., a smartphone, tablet, etc.). The promptrequesting user selection of one or more autonomous driving features totest may be presented to the user prior to entering the vehicle (e.g.,on the user's mobile device 130 before entering the vehicle, on theuser's laptop computer 160 at home, etc.). The prompt may additionallyor alternatively be presented to the user while the user is inside thevehicle 110, such as via a display screen of the vehicle, via a speakerin the vehicle, via the user's mobile device 116, etc.

In step 210, the computing device may receive a user selection of one ormore autonomous driving features to test. The user may provide his orselection via, for example, physical buttons or controls (e.g., via akeyboard, via vehicle buttons or control dials, etc.), touchscreenselections, spoken commands, and the like. The computing device maystore the user's selection in memory.

In some aspects, the driver can select the times and/or frequency thatthe driver prefers to be tested (or not to be tested). For example, thedriver may indicate that the driver would not like to be tested for thenext 30 minutes. The driver may indicate that the driver would like tobe tested from the hours of 4 to 6 PM, 7 to 8 AM, or any othertimeframe. The driver may also select the frequency of test, such aseach day, each month, twice a day, etc. In some aspects, the computingdevice may activate or deactivate the option for the driver to selectthe times and/or frequency for testing.

In step 215, the computing device may request and/or receive dataindicative of a driver's past driving telematics (e.g., drivinghistory). The data may be used to customize the tests recommended forthe driver based on past driving telematics, such that the test may bemore relevant to the driver. The driving telematics data may be storedin one or more databases (e.g., driving data source 140 or a database152), which may store a profile associated with the driver. The drivingtelematics may indicate which areas the driver may require morepractice. For example, the driving telematics may indicate that thedriver maintains a safe distance from vehicles in front of the driverless than a threshold amount of time (e.g., less than 50% of the time).Accordingly, the computing device may recommend that the driver betested on maintaining a safe distance from the vehicle in front by, forexample, disabling automatic speed control and instructing the driver tomaintain a certain distance from the vehicle in front of the driverduring a test. As another example, the driving telematics data mayindicate that the driver historically touches a lane marking with one ormore wheels on sharp curves more than 40% of the time. Accordingly, thecomputing device may recommend that the driver be tested on stayingwithin a driving lane on a sharp curve by, for example, disablingautomatic lane centering and instructing the driver to stay within thelane during a test.

The computing device may additionally or alternatively receive dataindicative of driving telematics across a population of drivers (e.g.,drivers in the same demographics and/or geographical region as thedriver to the tested). The telematics data of a population of driverscorresponding to the driver to be tested may be based on, for example,an indication of the driver's demographics and/or geographical regionstored in the driver's profile. The driving telematics for a populationof drivers may indicate which areas the population of drivers mayrequire more practice. For example, the driving telematics may indicatethat the population of drivers, on average, maintain a safe distancefrom vehicles in front of a particular driver less than a thresholdamount of time (e.g., less than 60% of the time). Accordingly, thecomputing device may recommend that the driver, who is part of thepopulation of drivers, be tested on maintaining a safe distance from thevehicle in front by, for example, disabling automatic speed control andinstructing the driver to maintain a certain distance from the vehiclein front of the driver during the test. As other example, the drivingtelematics data for a population of drivers may indicate that thedrivers, on average, historically touch a lane marking with one or morewheels on sharp curves more than 50% of the time. Accordingly, thecomputing device may recommend that the driver, who is part of thepopulation of drivers, be tested on staying within a driving lane on asharp curve by, for example, disabling automatic lane centering andinstructing the driver to stay within the lane during a test. In someaspects, the population of drivers may comprise drivers that have drivenon a particular section of road, such as a dangerous intersection,stretch of freeway, lane merge area, etc. The data indicative of drivingtelematics across a population of drivers may be stored in one or moredatabases (e.g., driving data source 140 or database 152).

In step 220, the computing device may receive data indicative of one ormore driving situations likely to lead to an event, such as an insuranceevent. In some aspects, a risk map may associate, for example,particular portions of a road with one or more risk scores. The riskscores may indicate the likelihood of an insurance event occurring atthe particular portion of road. In some aspects, the computing devicemay determine to test the driver in situations, roads, and/or speedssimilar to one or more of the portions of road in the risk map having ahigher risk score (e.g., greater than a threshold risk score). The dataindicative of one or more driving situations likely to lead to an eventmay be stored in one or more databases (e.g., driving data source 140 ordatabase 152).

In step 225, the computing device may generate one or more test(s) forthe driver based on the received data, such as the user's preferences,the user's past driving telematics, driving telematics for a populationof drivers, and/or driving situations likely to lead to an event. Thecomputing device may generate a weight for each factor for determiningthe one or more tests. For example, if there are four factors (e.g.,factor A, factor B, factor C, and factor D) based on the received data,the computing device may give factor A (e.g., the user's preferences) aweight of 1, and the remaining four factors a weight of 0. Accordingly,the computing device may select one or more driving test(s)corresponding to factor A. As another example, the computing device maygive factor A a weight of 0.25, factor B (e.g., the user's past drivingtelematics) a weight of 0.25, factor C (e.g., driving telematics for apopulation of drivers) a weight of 0.25, and factor D (e.g., drivingsituations likely to lead to an event) a weight of 0.25. Accordingly,the computing device may select one or more driving test(s)corresponding to each of the four factors. For example, the computingdevice may select the top two driving tests (based on weight), the topfour driving tests, etc. Any other type and/or combination of weightsmay be used for each factor.

As previously described, each test may involve disabling one or moreautonomous driving features so that the driver can be tested and/ortrained on the missing autonomous driving feature(s). The test may alsocomprise a corresponding time period (e.g., 4 to 6 PM) and/or acorresponding frequency (e.g., every 2 weeks), as previously described.The autonomous driving feature(s) to be disabled, the time period(s),and/or the testing frequency may be stored by the computing device inone or more databases (and/or in association with one or more driverprofiles) for future retrieval (e.g., in database 152). In some aspects,the driver may practice the driving test(s) on a simulator or in anautonomous vehicle, as will be described in further detail below.

FIG. 3 is a flow diagram illustrating an example method of guidance,control, and testing of autonomous vehicle features and a driver'sresponse thereto according to one or more aspects of the disclosure. Thesteps illustrated in FIG. 3 may be performed by one or more of thedevices illustrated in FIG. 1.

In step 305, a computing device (e.g., of the vehicle 110) may activateone or more autonomous driving feature of the vehicle 110. Non-limitingexamples of autonomous driving features include autonomous speedcontrol, autonomous steering, autonomous braking, or any other type ofautonomous driving feature. In some aspects, the driver may initiate anautonomous feature of the vehicle. For example, the driver may initiatecruise control, distance control, and/or lane keeping. In response, thevehicle may activate the selected autonomous driving feature. In someaspects, the vehicle may automatically activate one or more autonomousfeature of the vehicle without user selection. For example, the vehiclemay initiate brake support, lane keeping, lane changing, and obstacledetection when the vehicle is started, when the vehicle begins driving,or at any other point while the driver is in the vehicle.

In step 310, the computing device may determine whether to initiate oneor more autonomous driving feature tests. Whether to initiate a test maydepend on various factors, such as whether the driver indicates a desireto be tested, the current time or time period relative to testing timeperiods, current driving conditions, and/or the current location of thevehicle (including the type of road the vehicle is on or approaching).

In some aspects, the driver may indicate a desire to be tested byproviding input to the vehicle 110 or a computing device within thevehicle (e.g., a mobile device 116). For example, the driver may give aspoken command to the vehicle 110 and/or a mobile device 116 to initiatea test, and the vehicle and/or mobile device may receive the command viaa microphone. The command may indicate a test selected by the driver,such as a test to maintain a safe distance from the vehicle in front ora test to practice merging with freeway traffic. The vehicle 110 maydetermine to initiate the test, based on the user command to initiatethe test. The driver may provide similar inputs via, for example, atouchscreen display, control buttons or dials, or any other inputdevices to initiate the test.

In some aspects, the computing device may initiate a test based on thecurrent time. As previously explained, the computing device may store atime, time frame, and/or frequency for each test in a database (e.g.,database 152). The computing device (e.g., a vehicle 110 computer) maycompare the current time to the time, time frame, and/or frequencystored in the database. If the current time corresponds to the time,time frame, and/or frequency stored in the database (e.g., if thecurrent time is within a threshold distance from the stored time, iswithin the time frame, etc.), the computing device may determine toinitiate a driving test.

In some aspects, the computing device may initiate a test based oncurrent driving conditions and/or the current location of the vehicle(including the type of road the vehicle is on or approaching). Forexample, the computing device may determine that the driver has enteredor will soon enter a ramp to merge onto another road (e.g., a freeway).The computing device may determine that the driver is to be tested onmerging on to the freeway, and in response, may initiate the test forthe driver to be tested on merging because the vehicle detected anupcoming merge situation. For example, the computing device may access amap that includes a plurality of roadway features, identify one or moremerge areas, and initiate the merging test as the vehicle approaches theidentified merge area (e.g., within a threshold distance from themerge). As another example, the computing device may use cameras orsensors of the vehicle to identify an upcoming merge (e.g., based onimage recognition of a street sign indicating an upcoming merge,identification of an exit-only lane (e.g., exit-only sign or differentlane markings, such as a solid line or more-frequent dash lines), andthe like). As another example, the computing device may detect thatanother vehicle (whether it signaled or not) is pulling into the samelane (and ahead) of the current vehicle. In response, the computingdevice may initiate a test for the driver to take over control andmaintain a certain distance from the vehicle in front (e.g., 3 seconddistance, 300 feet, etc. depending on the circumstance). The computingdevice may use this test to gauge the reaction time of the driver andmeasure, for example, how long it takes the driver to adjust thedistance to the vehicle in front. The computing device may initiate thetest even if the autonomous functionalities of the vehicle are able toadjust the distance to that vehicle. As yet another example, thecomputing device may detect that the vehicle is approaching a corner orcurve in the road. The computing device, instead of using autonomousfeatures to round the corner or curve, could instead initiate a test togauge the reaction time of the driver. Prior to the curve, the computingdevice may signal for the driver to take over driving and measure theamount of time for the driver to adjust the vehicle's speed anddirection (or otherwise respond to the test). The vehicle may use itsautonomous features to maintain a safe speed and control until thedriver assumes control.

In some aspects, there may be situations with parameters that are withinthe parameters of a vehicle's autonomous driving functionalitiesabilities. However, a change of one or more of the parameters couldresult in an event where the driver would be required to take overcontrol in certain autonomous modes. The computing device may givepreference to these types of situations, alerting the driver to takecontrol of the vehicle, and measure, for example, the amount of time forthe driver to take control and whether the driver performed the correctresponse to the situation.

In some situations, the computing device may determine that drivingconditions and/or the location of vehicle are good for simulating one ormore events and having the driver react to the one or more events. Thatis, while autonomous driving functions are in use, the computing devicemay simulate an event using virtual or augmented reality (e.g., via ahead-up-display). For example, the computing device may detect that thevehicle is on or is approaching a stretch of road that is straight withno significant curves or elevation changes, no speed limit changes, notraffic signals or stop signs, and/or no vehicular traffic in eitherdirection. The computing device may determine to create one or moreevents using, for example, virtual reality or augmented reality methodsin order to test the driver. For example, the vehicle may include ahead-up display and may simulate an obstacle (e.g., an animal, such as adeer) in the road by projecting an image of the obstacle onto thehead-up-display. The computing device may verify that the operator takesappropriate action, such as applying the brakes or switching lanes toavoid hitting the simulated obstacle. The computing device may similarlysimulate, for example, a slowing vehicle and detect the driver'sresponse to the slowing vehicle.

If the computing device determines not yet to initiate an autonomousdriving feature test (step 310: N), the computing device may wait untila test is to be initiated. For example, the current time or time periodmight not correspond to a testing time period, the current drivingconditions might not be amenable to testing, the vehicle might not yethave approached a roadway feature appropriate for a particular type oftest (e.g., a bend in the road to test the driver's ability to navigatethrough the bend, two or more lanes to test the driver's ability tochange lanes, a merge area to test the driver's ability to merge onto adifferent road, etc.). If, on the other hand, the computing devicedetermines to initiate an autonomous driving feature test (step 310: Y),the computing device may proceed to step 315.

In step 315, the computing device may select an autonomous drivingfeature test. The test may comprise one autonomous driving feature todisable during the test, or the test may comprise a plurality ofautonomous driving features to disable during the test. As previouslydescribed, each test may correspond to one or more driving feature todisable. The computing device may determine one or more driving featureto disable based on the selected test. In some aspects, the vehicle maygive the driver an option to select the test. In some aspects, thevehicle might not give the driver the option to select the test, but mayinstead select the test based on driving data. The vehicle may also givethe driver the option to cancel the test prior to initiating the test.

In step 320, the computing device may generate an indication to thedriver of the vehicle that one or more autonomous driving feature testswill begin. For example, the vehicle 110 or a mobile device 116 thereinmay provide, via a speaker of the vehicle or mobile device, an audibleindication to the driver that a test will begin, when the vehicle willdisable the one or more autonomous features, which features will bedisabled, and/or which test was selected. The vehicle may additionallyor alternatively display one or more indication via a display devicewithin the vehicle 110. The indication may include an alert that one ormore autonomous feature will be disabled. The alert to the driver may bethe same type of alert (or the same alert) as an alert to the driver areal scenario where the driver is required to take over manual control.For example, the alert may comprise flashing lights in the vehicle, aloud horn, or other audio or visual indications for the driver to takeover manual control. In some aspects, the computing device may generateand speak or display to the driver instructions associated with thetest, such as a plurality of tasks for the user to perform during thetest.

In step 325, the computing device may deactivate or adjust parameters ofthe one or more autonomous driving features associated with the selectedtest. In some aspects, the computing device may wait for a response(e.g., a confirmation) from the driver to begin the test beforedeactivating or adjusting parameters of any autonomous driving features.The computing device may receive the response from the driver inresponse to generating the indication in step 320. If the driver is tobe tested on merging on to a freeway, the vehicle may deactivate oradjust parameters of autonomous lane changing and/or autonomous speedcontrol so that the driver can practice merging on to the freeway. Asanother example, if the driver is to be tested on maintaining a safedistance from another vehicle, the vehicle may deactivate or adjustparameters of autonomous speed control so that the driver can practicemaintaining a certain distance from the vehicle in front.

In some aspects, adjusting parameters for one or more autonomous featuremay comprise adjusting the parameters to be within a particular margin(e.g., safety margin) while being outside of a comfortable range for thedriver. The driver may be expected to react by manually bringing thevehicle back into the correct range. The computing device may measurehow the driver reacted to the simulated issue, e.g., from the time thethe vehicle announces the need for the driver to take over.

In step 330, the computing device may generate, for indication to thedriver, instructions associated with the test. The instructions may begenerated and/or outputted to the driver, via one or more displaydevices or speakers, before initiating the test and/or during the test.Thus, while some autonomous driving functions are in use, the driver maybe instructed or alerted to resume control of the vehicle. Theinstructions may indicate to the driver an end goal, such as merging onto an upcoming freeway. Additionally or alternatively, the instructionsmay indicate step-by-step instructions. For example, the computingdevice may first indicate to the driver approaching a merge ramp to slowdown to the speed limit of the merge ramp. As the driver approaches thefreeway to merge on to, the computing device may indicate to the driverto match the speed of traffic on the freeway. As the driver prepares tomerge on to the freeway, the computing device may indicate to the driverto move over to the next lane of the freeway.

In step 335, the computing device (e.g., drive test computer 151, avehicle 110 computing device, and/or a mobile device 116) may measurethe driver's response to the test. The computing device may receivesensor data from vehicle sensors 119 and/or sensors 118 of the mobiledevice to measure the driver's response. As previously described, thesensor data may indicate the path of travel of the vehicle, vehiclespeed, vehicle acceleration, whether the vehicle stayed within certainlanes, the direction of travel of the vehicle, etc. Examples of driverresponses to the test include, for example, the driver adjusting thedistance to the vehicle in front, adjusting the speed of the vehicle,adjusting a direction of the vehicle (e.g., a turn, going around acurve, a lane change, etc.), applying brakes, etc. The computing devicemay measure how long it takes the driver to respond to deactivation oradjustment of the one or more autonomous driving feature and/or aninstruction (e.g., alert) to the driver to take over control of thevehicle. For example, the vehicle may instruct the driver to move to theright-most lane and measure the duration of time from providing theinstruction to the driver to the driver initiating actions to move tothe right-most lane (e.g., by turning the steering wheel to the right)or to the vehicle actually entering the right-most lane. The vehicle mayadditionally or alternatively measure duration of time from deactivatingautonomous lane changing to the driver initiating actions to move to theright-most lane or to the vehicle entering the right-most lane, such asif the time of instruction and the time of deactivating one or moreautonomous features associated with the test is different.

In some aspects, the computing device may determine how the driverresponded to the instruction and/or deactivation (or adjustment) of theautonomous driving feature(s) (e.g., actions taken by the driver). Inthe lane change example above, the computing device may determine, forexample, whether the driver used the vehicle's turn signal whileswitching lanes and/or prior to switching lanes. The computing devicemay determine whether or not the driver applied brakes during the lanechange operation. If the vehicle or mobile device includes a camera(e.g., a camera with facial recognition and/or infrared capabilities todetermine the direction that the driver's head or eyes is facing orother movement of the driver's head or eyes), the computing device maydetermine whether the driver turned to check the side-view mirror beforechanging lanes. The computing device may also determine whether (andwhen) the driver has completed the operation (e.g., switched to theright-most lane). The computing device may also measure, for example,any “jerkiness” caused by the driver when assuming control and/or whilecompleting the test.

Data logs of the generated tests (e.g., simulated events) and thedriver's response thereto (e.g., response time and actions) may berecorded and stored at a data storage location of the vehicle 110 ormobile device 116 and/or transferred to another data storage location,such as a remote storage location (e.g., autonomous vehicle drive testserver 150). The remote storage location may be associated (e.g., owned,operated, or used) by an outside company, such as the vehiclemanufacturer or an insurer of the vehicle or driver. The data may betransferred to the other data storage location at any time, and the timeof transfer may be based on cell coverage of the vehicle or the mobiledevice, network costs, etc. The data logs may be used to improve thefunctionality of the drive testing system and to verify the driver'sresponses to the test scenarios. The driver may be permitted to reviewhis or her performance on various tests and compare his or her responsesto a population of other drivers (e.g., with appropriate anonymizationof the other driver's responses).

In step 340, the computing device may determine whether the test iscomplete or to otherwise end the test. If not (step 340: N), thecomputing device may return to step 330 and/or step 335 to generateinstructions for the test and/or measure the driver's response to thetest. If the computing device determines to end the test (step 340: Y),the computing device may proceed to step 345. The computing device maydetermine to end the test in one or more scenarios. In some aspects, thecomputing device may end the test in response to receiving a commandfrom the driver to end the test. As another example, the computingdevice may end the test once the driver completes the test. In the lanechange example, the computing device may end the test after the vehiclehas switched to the right-most lane. In a freeway merge test, thecomputing device may end the test once the vehicle has completed itsmerge onto the freeway. In some aspects, in order for autonomousfunctionality to take over driving, the driver may perform several stepsand the vehicle may detect the driver performing these steps. Forexample, the driver may signal to the vehicle for autonomousfunctionality to take over driving, the autonomous functionality maysignal to the driver that the vehicle is able to take back control, andthe autonomous functionality may go live when the driver quits touchingthe steering wheel, pedals, or similar.

In some aspects, the computing device may end the test if the driverdoes not respond quickly enough to the instruction and/or deactivationof an autonomous feature (e.g., if it takes the driver longer than athreshold amount of time to response). For example, if the vehicleinstructs the driver to move one lane to the left, and the vehicledetermines that the driver has failed to initiate moving the vehicle onelane to the left within 5 seconds of the instruction (e.g., by graspingthe wheel, turning the wheel to the left, initiating the left turnblinker, turning the driver's head to view the side-view mirror, etc.),the computing device may end the test. As another example, if thevehicle deactivates an autonomous feature (e.g., autonomous lanechanging), and the vehicle determines that the driver has failed toinitiate moving the vehicle one lane to the left within 1 second of thedeactivating autonomous lane changing, the computing device may end thetest.

In step 345, the computing device may reactivate the autonomous drivingfeature(s) deactivated for the test or may readjust the parametersadjusted for the test (e.g., in step 325). That is (and in someaspects), the computing device may reactivate one or more autonomousdriving features after the test is ended. For example, if autonomousbraking and autonomous speed control were deactivated for the test, thecomputing device may reactivate autonomous braking and autonomous speedcontrol after the test ends.

In some aspects, the computing device may provide feedback to the driverafter the test. Examples of feedback to driver may include an indicationthat the test is complete, that data for the test was recorded, how thedriver responded (e.g., as compared to nationally or another populationgroup), or any other type of feedback. The computing device may providethe feedback to the driver while the driver is still in the vehicle 110or after the driver exits the vehicle. The feedback may be provided tothe driver via, for example, a speaker and/or a display device inside oroutside of the vehicle 110. In some aspects, the driver may receive adetailed test report. The computing device may provide feedback todrivers on their abilities to react to certain situations, and may focuson improving the driver's reaction in situations where the user mighthave been slow to respond. The computing device may also providefeedback to maintain the driver's skills in areas where the driverdemonstrated a high level of skill. In some aspects, the feedback maycomprise video of the driver's own reactions with graphics of theenvironment. The feedback may also include video of how other drivershandled similar situations and where the driver might need to improve.

In step 350, the computing device may generate one or more scores forthe driver based on the driver's driving response(s) during the test,such as actions taken by the driver and/or the amount of time for thedriver to respond (e.g., reaction time). The score may be indicative of,for example, how safely the driver performed the test or a particularmaneuver. In some aspects, the score may factor in the amount of timefor the driver to respond, such as to take over control (e.g., grab thesteering wheel), initiate a maneuver, or complete a maneuver or test.The driver's score may increase the less time it took the driver toinitiate or complete the maneuver or test and may decrease the more timethe driver took. For example, assume that the test was a test to switchone lane over. The computing device may determine the driving score forthat particular test to be 5 if the driver took between 3 seconds and 6seconds to complete the lane change, 4 if the driver took between 6seconds and 7 seconds to complete the lane change, and 3 if the drivertook more than 7 second to complete the lane change. Each type ofmaneuver and/or test may be associated with a plurality of scoresindicative of the time for the driver to initiate or complete themaneuver and/or test.

In some aspects, each type of maneuver and/or test may be associatedwith a plurality of scores indicative of the actions taken by the driverto complete the maneuver. For example, each test may include a pluralityof checklist items. The computing device may factor, in to the drivingscore, the actions taken by the driver to complete the maneuver. Forexample, assume again that the test was a test to switch one lane over.The checklist items for this test may include (1) use of the correctturn signal before crossing the lane marking, (2) the driver checkingthe side-view mirror, (3) the driver checking a blind spot (ifapplicable), (4) use of the correct turn signal while crossing the lanemarking, and (5) turning off the turn signal once the vehicle iscompletely in the new lane. Each item on the checklist may contribute tothe driver score. For example, if the driver performed all five itemsduring the test, the driver may receive a score of 5 for the test (e.g.,1 for each item). If the driver performed four out of five items duringthe test, the driver may receive a score of 4, and so on. In theseexample, each item is weighted the same (e.g., 1). However, the itemsmay be weighted differently depending on the importance of each item.For example, use of the turn signal before crossing the lane marking maybe weighted higher than turning off the turn signal once the vehicle iscompletely in the new lane. In some aspects, an item on the checklistmay have more than two values. For example, assume that the checklist inthe above example includes the angle (e.g., maximum angle) of thesteering wheel during the lane change. If the driver was between angle Aand angle B, the driver may receive a score of 0.5 for this item. If thedriver was between angle B and angle C, the driver may receive a scoreof 1 for this item. If the driver was between angle C and angle D, thedriver may receive a score of 1.5, and so on.

As previously discussed, the computing device may generate reportsshowing how the driver's reaction time and/or skills compare to otherdrivers and to industry standard normative data in the same or similarcircumstances or tests. With a large population of users, driving scorescould be correlated to insurance losses, with resultant improvements incorrectly charging vehicle drivers for their skills, instead of chargingfor items which show correlation to loss.

In step 355, the computing device may adjust a property of insurance forthe driver based on the driver's response during the test. For example,based on the amount of time it took the driver to respond and/or theactions taken by the driver, the computing device may increase ordecrease the driver's insurance rate. The insurance rate may increase ifthe driver's maneuvers were unsafe (e.g., a lower driver score) ordecrease if the driver's maneuvers were safer (e.g., a higher driverscore). In some aspects, an insurance rate may be increased or decreasedfor the current drive and may reset to the original insurance rate afterthe current drive ends. The driver and/or vehicle may also be switchedover to higher-priced catastrophe insurance if the driver's score wasparticularly low. In some aspects, the driver may lose insurance basedon unsafe actions taken during a test (or over several tests).

In step 360, the computing device may determine whether the driver'sresponse during the test satisfies a threshold. If so (step 360: Y), thecomputing device may return to step 310 to determine whether theinitiate a different autonomous driving feature test for the driver. Forexample, if the driver's score is greater than a particular threshold,the computing device may determine that the driver passed the test(e.g., safely responded to the driving scenario) and return to step 310to determine whether to initiate another test for the driver. If, on theother hand, the driver's response does not satisfy the threshold (step360: N), the computing device may proceed to step 365.

In step 365, the computing device may determine whether to repeat thetest. The computing device may request that the driver repeat the testif, for example, the driver's score was below a threshold value. Forexample, if the driver's reaction time is above a threshold time, thedriver may be tested more because the driver might not know a particularmaneuver well. The driver may be tested during the same drive (e.g., ifthe computing device detects a similar scenario where the test would beproper) or in a future drive. If the computing device determines not torepeat the test (step 365: N), the computing device may return to step310 to determine whether the initiate a different autonomous drivingfeature test for the driver. If the computing device determines torepeat the test (step 365: Y), the computing device may return to step320 and reinitiate the same autonomous driving feature test. In someaspects, if the driver's reaction time (or other response) is above athreshold, the computing device may disable one or more autonomousdriving features for the current drive or for future drives because thedriver might not know how to respond if the driver has to take overmanual control during certain circumstances. The computing device maynotify the driver of the disabled features and how long the featureswill be disabled.

By providing events where the driver is requested to assume control ofthe vehicle, the system may teach the driver to react to a change ofcontrol (e.g., from autonomous to manual) with practice and without apossibly dangerous knee jerk reaction. By the system staging each eventrequesting the driver to assume control to react to a real or virtualevent outside the vehicle, the system may teach the driver to quicklyscan the environment, create and quickly evaluate a mental map of thevehicle in space and the hierarchy of issues the driver may face. Thesystem may improve the safety of the handoff of control to the driver,particularly in situations where the autonomous driving functionalitymight not be able to make sense of its inputs and the event is a reallife event, not a drill. The logging of driver performance in real worldevents requesting the driver to assume control of the vehicle may alsobe a method that manufacturers, industry safety trade groups, orinsurers to use to develop new scenarios or tests (or refine currentscenarios or tests) that better represent situations that autonomoussystems might not be able to understand and which might require driversto handle.

FIG. 4 illustrates a block diagram of a computing device 401 inautonomous vehicle drive test system 400 that may be used according toone or more illustrative embodiments of the disclosure. The computingdevice 401 may have a processor 403 for controlling overall operation ofthe computing device 401 and its associated components, including RAM405, ROM 407, input/output module 409, and memory unit 415. Thecomputing device 401, along with one or more additional devices (e.g.,terminals 441, 451) may correspond to any of multiple systems ordevices, such as systems or devices for guidance, control, and testingof autonomous vehicle features and a driver's response thereto.

Input/Output (I/O) module 409 may include a microphone, keypad, touchscreen, and/or stylus through which a user of the computing device 401may provide input, and may also include one or more of a speaker forproviding audio input/output and a video display device for providingtextual, audiovisual and/or graphical output. Software may be storedwithin memory unit 415 and/or other storage to provide instructions toprocessor 403 for enabling device 401 to perform various functions. Forexample, memory unit 415 may store software used by the device 401, suchas an operating system 417, application programs 419, and an associatedinternal database 421. The memory unit 415 includes one or more ofvolatile and/or non-volatile computer memory to storecomputer-executable instructions, data, and/or other information.Processor 403 and its associated components may allow the computingdevice 401 to execute a series of computer-readable instructions toreceive data from vehicles, mobile devices, and/or driving data sources,and using the data to measure the driver's response to driving tests.

The computing device 401 may operate in a networked environment 400supporting connections to one or more remote computers, such asterminals/devices 441 and 451. The computing device 401, and relatedterminals/devices 441 and 451, may include devices installed invehicles, mobile devices that travel within vehicles, or devices outsideof vehicles that are configured to receive and process sensor and otherdata. Thus, the computing device 401 and terminals/devices 441 and 451may each include personal computers (e.g., laptop, desktop, or tabletcomputers), servers (e.g., web servers, database servers), vehicle-baseddevices (e.g., on-board vehicle computers, short-range vehiclecommunication systems, sensors and telematics devices), or mobilecommunication devices (e.g., mobile phones, portable computing devices,and the like), and may include some or all of the elements describedabove with respect to the computing device 401. The network connectionsdepicted in FIG. 4 include a local area network (LAN) 425 and a widearea network (WAN) 429, and a wireless telecommunications network 433,but may also include other networks. When used in a LAN networkingenvironment, the computing device 401 may be connected to the LAN 425through a network interface or adapter 423. When used in a WANnetworking environment, the device 401 may include a modem 427 or othermeans for establishing communications over the WAN 429, such as network431 (e.g., the Internet). When used in a wireless telecommunicationsnetwork 433, the device 401 may include one or more transceivers,digital signal processors, and additional circuitry and software forcommunicating with wireless computing devices 441 (e.g., mobile phones,short-range vehicle communication systems, vehicle sensing andtelematics devices) via one or more network devices 435 (e.g., basetransceiver stations) in the wireless network 433.

It will be appreciated that the network connections shown areillustrative and other means of establishing a communications linkbetween the computers may be used. The existence of any of variousnetwork protocols such as TCP/IP, Ethernet, FTP, HTTP and the like, andof various wireless communication technologies such as GSM, CDMA, Wi-Fi,and WiMAX, is presumed, and the various computing devices and autonomousvehicle drive test system components described herein may be configuredto communicate using any of these network protocols or technologies.

Additionally, one or more application programs 419 used by the computingdevice 401 may include computer executable instructions (e.g.,algorithms for guidance, control, and testing of autonomous vehiclefeatures and a driver's response thereto) for initiating one or moreautonomous driving feature tests and measuring the driver's response,and performing other related functions as described herein.

While the aspects described herein have been discussed with respect tospecific examples including various modes of carrying out aspects of thedisclosure, those skilled in the art will appreciate that there arenumerous variations and permutations of the above described systems andtechniques that fall within the spirit and scope of the invention.

What is claimed is:
 1. A system comprising: a first computing device anda second computing device, wherein the first computing device comprises:one or more sensors configured to measure driving data associated withan autonomous vehicle; a transmitter configured to transmit driving datameasured by the one or more sensors; a first processor; and first memorystoring computer-executable instructions that, when executed by thefirst processor, cause the first computing device to: activate aplurality of autonomous driving features of the autonomous vehicle;determine a location of the autonomous vehicle; based on the location ofthe autonomous vehicle, initiate a driving test; based on initiating thedriving test, deactivate or adjust parameters of one or more of theplurality of autonomous driving features; receive, from the one or moresensors and during the driving test, driving data associated with theautonomous vehicle; and send, via the transmitter, the driving dataassociated with the autonomous vehicle to the second computing device,wherein the second computing device comprises: a receiver; a secondprocessor; and second memory storing computer-executable instructionsthat, when executed by the second processor, cause the second computingdevice to: receive, via the receiver, the driving data associated withthe autonomous vehicle; and based on the driving data associated withthe autonomous vehicle, determine a response time of a driver of theautonomous vehicle and actions taken by the driver during the drivingtest.
 2. The system of claim 1, wherein the plurality of autonomousdriving features two or more of autonomous speed control, autonomoussteering, or autonomous braking.
 3. The system of claim 1, wherein thefirst memory stores computer-executable instructions that, when executedby the first processor, cause the first computing device to: compare acurrent time to a time associated with the driving test, whereininitiating the driving test is based on the location of the autonomousvehicle and comparing the current time to the time associated with thedriving test.
 4. The system of claim 1, wherein the driving testcomprises a test to merge on to a road, and wherein the deactivated oradjusted parameters of one or more of the plurality of autonomousdriving features comprises at least autonomous lane changing andautonomous speed control.
 5. The system of claim 1, wherein the drivingtest comprises a test to maintain a safe distance from another vehiclein front of the autonomous vehicle, and wherein the deactivated oradjusted parameters of one or more of the plurality of autonomousdriving features comprises at least autonomous speed control.
 6. Thesystem of claim 1, wherein the second memory stores additionalcomputer-executable instructions that, when executed by the secondprocessor, cause the second computing device to: generate a data logcomprising an indication of the driving test, the response time, and theactions taken by the driver during the driving test.
 7. The system ofclaim 1, wherein the first memory stores computer-executableinstructions that, when executed by the first processor, cause the firstcomputing device to: based on a determination that the driver completedthe driving test or a determination that the response time exceeds athreshold time, reactivate the deactivated or readjust the adjustedparameters of one or more of the plurality of autonomous drivingfeatures.
 8. The system of claim 1, wherein the second memory storescomputer-executable instructions that, when executed by the secondprocessor, cause the second computing device to: based on the responsetime and the actions taken by the driver during the driving test,performing one or more of the following: generate a drive score for thedriver; or adjust a property of insurance for the driver or theautonomous vehicle.
 9. A non-transitory computer readable medium storinginstructions that, when read by a computing device, cause the computingdevice to: activate a plurality of autonomous driving features of anautonomous vehicle; determine a location of the autonomous vehicle;based on the location of the autonomous vehicle, initiate a drivingtest; based on initiating the driving test, deactivate or adjustparameters of one or more of the plurality of autonomous drivingfeatures; receive, during the driving test and from one or more sensorsof the autonomous vehicle or one or more sensors of a mobile computingdevice within the autonomous vehicle, driving data associated with theautonomous vehicle; and based on the driving data associated with theautonomous vehicle, determine a response time of a driver of theautonomous vehicle and actions taken by the driver during the drivingtest.
 10. The non-transitory computer readable medium of claim 9,wherein the plurality of autonomous driving features comprise two ormore of autonomous speed control, autonomous steering, or autonomousbraking.
 11. The non-transitory computer readable medium of claim 9,storing instructions that, when read by the computing device, cause thecomputing device to: compare a current time to a time associated withthe driving test, wherein initiating the driving test is based on thelocation of the autonomous vehicle and comparing the current time to thetime associated with the driving test.
 12. The non-transitory computerreadable medium of claim 9, wherein the driving test comprises a test tomerge on to a road, and wherein the deactivated or adjusted parametersof one or more of the plurality of autonomous driving features comprisesat least autonomous lane changing and autonomous speed control.
 13. Thenon-transitory computer readable medium of claim 9, wherein the drivingtest comprises a test to maintain a safe distance from another vehiclein front of the autonomous vehicle, and wherein the deactivated oradjusted parameters of one or more of the plurality of autonomousdriving features comprises at least autonomous speed control.
 14. Thenon-transitory computer readable medium of claim 9, storing instructionsthat, when read by the computing device, cause the computing device to:generate a data log comprising an indication of the driving test, theresponse time, and the actions taken by the driver during the drivingtest.
 15. The non-transitory computer readable medium of claim 9,storing instructions that, when read by the computing device, cause thecomputing device to: determine that the driver failed to initiate,within a threshold time, at least one driving operation of a pluralityof driving operations associated with the driving test.
 16. Thenon-transitory computer readable medium of claim 9, storing instructionsthat, when read by the computing device, cause the computing device to:based on the response time and the actions taken by the driver duringthe driving test, performing one or more of the following: generate adrive score for the driver; or adjust a property of insurance for thedriver or the autonomous vehicle.
 17. A method, comprising: activating,by a computing device associated with an autonomous vehicle, a pluralityof autonomous driving features of the autonomous vehicle; determining alocation of the autonomous vehicle; based on the location of theautonomous vehicle, initiating a driving test; based on initiating thedriving test, deactivating or adjusting parameters of one or more of theplurality of autonomous driving features; receiving, during the drivingtest and from one or more sensors of the autonomous vehicle or one ormore sensors of a mobile computing device within the autonomous vehicle,driving data associated with the autonomous vehicle; and based on thedriving data associated with the autonomous vehicle, determining aresponse time of a driver of the autonomous vehicle and actions taken bythe driver during the driving test.
 18. The method of claim 17, whereinthe plurality of autonomous driving features comprise two or more ofautonomous speed control, autonomous steering, or autonomous braking.19. The method of claim 17, further comprising: comparing a current timeto a time associated with the driving test, wherein initiating thedriving test is based on the location of the autonomous vehicle and thecomparing the current time to the time associated with the driving test.20. The method of claim 17, wherein the driving test comprises a test tomerge on to a road, and wherein the deactivated or adjusted parametersof one or more of the plurality of autonomous driving features comprisesat least autonomous lane changing and autonomous speed control.